Tata Consultancy Services | Big Data Specialist | united states

Tata Consultancy Services • united states, united states, us • 1m ago

Job Description

Big Data (PySpark) Tech Lead

10 + Years Overall Experience in Data Management, Data Lake and Data Warehouse
6+ Years Hadoop, Hive, Sqoop, SQL, Teradata
6+ Years PySpark(Python and Spark), Unix
Good to have Industry leading ETL experience
Banking Domain experience

Key Responsibilities

Ability to design, build and unit test applications on Spark framework on Python.
Build PySpark based applications for both batch and streaming requirements, which will require in-depth knowledge on majority of Hadoop and NoSQL databases as well.
Develop and execute data pipeline testing processes and validate business rules and policies
Optimize performance of the built Spark applications in Hadoop using configurations around Spark Context, Spark-SQL, Data Frame, and Pair RDD's.
Optimize performance for data access requirements by choosing the appropriate native Hadoop file formats (Avro, Parquet, ORC etc) and compression codec respectively.
Ability to design & build real-time applications using Apache Kafka & Spark Streaming
Build integrated solutions leveraging Unix shell scripting, RDBMS, Hive, HDFS File System, HDFS File Types, HDFS compression codec.
Build data tokenization libraries and integrate with Hive & Spark for column-level obfuscation
Experience in processing large amounts of structured and unstructured data, including integrating data from multiple sources.
Create and maintain integration and regression testing framework on Jenkins integrated with BitBucket and/or GIT repositories
Participate in the agile development process, and document and communicate issues and bugs relative to data standards in scrum meetings
Work collaboratively with onsite and offshore team.
Develop & review technical documentation for artifacts delivered.
Ability to solve complex data-driven scenarios and triage towards defects and production issues
Ability to learn-unlearn-relearn concepts with an open and analytical mindset
Participate in code release and production deployment.
Challenge and inspire team members to achieve business results in a fast paced and quickly changing environment

Related Jobs

Uber | Drive with Uber

Uber • escondido, ca, us • 8h ago

8h ago

Headway | LMFT (Remote)

Headway • duluth, mn, us • 8h ago

8h ago

Uber | Drive with Uber - Earn at least $1886 in your first 148 trips with Uber.

Uber • waddell, az, us • 8h ago

8h ago

Uber | Choose your schedule - Earn At Least $1662 For Your First 134 Trips, Guaranteed.

Uber • setauket, ny, us • 8h ago

8h ago

Headway | Licensed Psychiatric NP

Headway • austin, tx, us • 8h ago

8h ago

MD Staff, LLC | Physician / Gastroenterology / Virginia / Locum Tenens / Locums/IM-Gastroenterology/Job/Virginia Job

MD Staff, LLC • lynchburg, va, us • 8h ago

8h ago

Uber | Looking For Part-Time Jobs? - Drive with Uber

Uber • accord, ny, us • 8h ago

8h ago

MD Staff, LLC | Physician / Surgery - Cardiothoracic / Tennessee / Locum Tenens / Locums/Cardiothoracic Surgery/Job/Tennessee Job

MD Staff, LLC • memphis, tn, us • 8h ago

8h ago

Mosaic | Home Provider - Iowa/Sioux County

Mosaic • rowan, ia, us • 8h ago

8h ago

Uber | Uber - Sign Up to Drive and Earn

Uber • heber, ca, us • 8h ago

8h ago